NTOU Presentation

Agent Based Models

• Define
  • Individuals
  • Interactions
• Observe Emergent Patterns

Traffic Example

• Robustness
• Heterogeneity and Mean
• Emergent Behaviour
• Policy Making

Traffic Example - 2

When should we bother?

• Knowledge is only available at individual level
• When aggregation is too hard to model mathematically
• When we want to test what would generate an aggregate pattern

Shelling model 1

Shelling model 2

Individual Level Policy - 1

Individual Level Policy - 2

Agent Based Models of Fisheries

• Exploration
• Ecosystems
• Complicated

Allen & McGlade (1986)

Cabral et al (2010)

Beecham and Engelhard (2007)

Gao and Hailu (2010)

Saul (2013)

Bastardie et al (2010)

Cenek and Franklin (2017)

Cooper and Jarre (2017)

POSEIDON

What do we want

• Policy Simulator
  • Agents Flexibility
  • Model Flexibility
• Final Objectives:
  • “Laboratory” to try new policies
  • Connect policy-maker objectives to strategies
  • Connect indicators with actions

The One Agent Problem

The One Agent Problem

• Find the most profitable spot to fish
• Constraints:
  • No biomass information
  • No model knowledge
  • Environment changes over time
• Subproblems:
  • How to explore
  • Explore-Exploit Tradeoff

Explore or Exploit ?

Explore or Exploit ?

Explore-Exploit

• Stochastically choose to explore next trip with probability \(p\)
• Explore in the neighborhood of where you currently go

One Agent world

One Agent sample run

Scaling issues

Scaling issues

Scaling issues

L’enfer, c’est les autres

• Other boats consume biomass
• You can use other boats information
• How to imitate?
• With probability \(p\) explore, otherwise copy somebody who’s doing better

Explore-Exploit-Imitate

# with probability epsilon, explore
if(runif()<epsilon)
  # shock your position by delta
  position <- position + runif(min=-delta,max=delta)
else
  # if a friend is doing better, play their slot machine
  if(profits[me] < profits[best_friend])
    position<- positions[best_friend]
  # otherwise play the previous slot machine
play(position)

Two Agents sample run

Many Agents

Cui prodest?

• Model free
• Adaptive

Oil Prices

Fish the Line (part 1)

Fish the Line (part 2)

A flexible simulator

• Flexible in terms of:
  • Decisions
  • Biology
  • Algorithms

Target Switching

Gear Selection

OSMOSE

West Coast

WFS

Gravitational Search

# peek at everyone's profits and normalize them
masses<- (profits-min_profits)/(max_profits-min_profits)
# single out the top Z most profitable competitors
best<-order(masses)[1:Z]
# accelerate as if driven by "gravity" towards the best
acceleration<- acceleration+ sum(g * masses[me]*masses[best]/distance(me,best))
acceleration<- acceleration/masses[me]
# move (with some friction)
velocity<- velocity * runif() + acceleration
position<-position + velocity
play(position)

Gravitational Search - Demo

Kernel Regression

Kernel Regression - Demo

Policies

Simulating Policies

• Open Loop
  • Scenario Evaluation
  • Policy Optimization
• Closed Loop
  • Policy Search
  • Policy Discovery

Open Loop

Scenario Evaluation

1. You have adaptive agents
2. Somebody hands you a set of policies to test
3. Apply each in turn
4. Check which performs best

TAC vs ITQ (mileage)

TAC vs ITQ (catchability)

Seventy-Thirty World

ITQ Prices

• Quotas are distributed 90% reds, 10% blue
• 

Blues are choke species

ITQ drives Gear (start)

ITQ drives Gear (end)

Gear fixes wastes

North-South world

Location choices

ITQ incentivates geography

Layered Policy - setup

Layered Policy - Anarchy

Layered Policy - MPA

Layered Policy - MPA + ITQ

Policy Optimization

1. You have adaptive agents
2. Somebody hands you a family of policies
3. You want to find the “best” parameters

Optimal MPA

• Geographically split world
• Find the single MPA that maximizes a score

Optimal MPA

\[ \text{Score} = \text{Blue Biomass}_{t=20} \]

Optimal MPA

\[ \text{Score} = \text{Blue Biomass}_{t=20} + \sum_{i=1}^{20} \text{Red Landings}_{t=i}\]

Optimal MPA - Well-mixed

Perfect Enforcement

No Enforcement

Decent Enforcement

• Hourly probability 15% caught
• 1000$ fine

Why do people cheat?

Optimal level of enforcement

• You want to maximize 20 years profits
• You can impose an MPA

Free Enforcement

• MPA is free and enforcement is perfect

Expensive Enforcement

• Enforcement costs: \[ 10M * p + 10000 * \text{MPA area} \]

Optimal Quotas

\[ \text{Score} = \text{Blue Biomass}_{t=20} + \sum_{i=1}^{20} \text{Red Landings}_{t=i}\]

• Geographically split map
• 300 fishers
• Very different quota values for TAC and ITQ

Optimal TAC

Optimal ITQ

Well-mixed world?

In a scenario where fishers are unable to respond to incentives the optimal quotas under TACs and ITQs are exactly the same

Pareto Front

Heterogeneous fleets

• 2 kinds of boat:
  • Small boats
  • Large boats
• 2 Objectives:
  • Maximize small boat income
  • Maximize efficiency
• 1 Policy lever:
  • Build MPA

Fairness Front

Right-most solution

Left-most solution

Closed Loop

Policy Search

1. You have adaptive agents
2. Somebody hands you an adaptive decision rule mapping state to actions
3. You want to find the “best” mapping parameters

Bluemania

• Well mixed world
• Want to incentivate gear change through a landing tax
• Blue fish worth 3 times red fish

No intervention

PID Taxation

• Expensive (blue) stock gets consumed too rapidly
• Geographically separated
• Update tax smoothly such that every day only about 600 units of blue stock is landed daily
• Poor man’s quotas
• Use a PI controller \[ p_{t+1} = a e_t + b \sum_{i=0}^T e_{i} \] \[ e_t = \text{Landings} - 600 \]
• “Autopilot” policy
• Parameters matter
• Noise matter

PID Taxation - demo

PID Taxation - optimal

Policy Discovery

1. You have adaptive agents
2. You have state (possibly degraded) indicators and action levers
3. You have to figure out how to link the two together
4. You want the decision rule to be optimal

Policy Discovery